For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as efficient as implicit differentiation for fast algorithms (e.g., superlinear

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Firstly, many machine learning models use optimization algorithms which require access to derivatives of the model.

Optimization is an integral part of most machine learning systems and most numerical optimization schemes relyonthecomputation ofderivatives.

While existing automatic differentiation (AD) frameworks allow flexibly composing model architectures, theydonotprovide thesame flexibility forcomposing learning algorithms--everything has to be implemented in terms of backpropagation.

Several machine learning applications involve the optimization of higher-order derivatives(e.g., gradients ofgradients) during training, which can beexpensive with respect to memory and computation even with automatic differentiation.

This approach allows for formal subdifferentiation: forinstance, replacing derivativesbyClarkeJacobians in the usual differentiation formulas is fully justified for a wide class of nonsmooth problems.